Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add metric for receiver block backfill failures #912

Merged
merged 1 commit into from
Jul 23, 2024

Conversation

darunrs
Copy link
Collaborator

@darunrs darunrs commented Jul 23, 2024

Adds a metric for tracking receiver block backfill status. Since the backfill has an end state, I can't rely on constantly incrementing a counter. Instead, I use a gauge and check for non zero values. If the backfill starts and completes without issue, the value will be 0. But, if the backfill fails, it will increment the gauge and then skip to Lake backfill, leaving the gauge at a nonzero value. We can alert on this. If the stream is restarted, then a successful attempt at the backfill will reset the gauge to 0.

@darunrs darunrs requested a review from a team as a code owner July 23, 2024 19:56
@darunrs darunrs changed the title feaat: Add metric for receiver block backfill failures feat: Add metric for receiver block backfill failures Jul 23, 2024
@darunrs darunrs linked an issue Jul 23, 2024 that may be closed by this pull request
@darunrs darunrs merged commit 38ac863 into main Jul 23, 2024
4 checks passed
@darunrs darunrs deleted the altering-on-bitmap-backfill branch July 23, 2024 23:45
@darunrs darunrs mentioned this pull request Jul 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Monitor and alert on bitmap indexer source failures
1 participant